BC5 readiness: TypeScript live wire-capture canary#294
Conversation
5bd407c to
63205bf
Compare
b7e9460 to
d776bfd
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7e946013c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
6 issues found across 12 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="conformance/runner/typescript/live-dispatch.ts">
<violation number="1" location="conformance/runner/typescript/live-dispatch.ts:106">
P2: Dispatch coverage check should not use `in` on a plain object; inherited keys can bypass missing-operation detection.</violation>
</file>
<file name="conformance/runner/typescript/schema-validator.ts">
<violation number="1" location="conformance/runner/typescript/schema-validator.ts:75">
P2: Response schema lookup ignores non-200 success responses (e.g. 201), causing false validation failures for valid operations.</violation>
<violation number="2" location="conformance/runner/typescript/schema-validator.ts:124">
P1: Compiling a response schema fragment leaves `#/components/...` refs without the OpenAPI root context, which can break AJV `$ref` resolution at runtime.</violation>
<violation number="3" location="conformance/runner/typescript/schema-validator.ts:182">
P2: `readObjectProperties` returns `null` for array response schemas, causing `collectExtras` to short-circuit and return `[]`. List endpoints (ListProjects, ListTodolists, ListTodos, etc.) that return top-level arrays will never report additive fields inside array items, silently missing BC5 forward-compat wire additions in the extras-observed summary. Consider unwrapping `items` for array schemas to inspect item-level properties.</violation>
</file>
<file name="conformance/runner/typescript/live-runner.test.ts">
<violation number="1" location="conformance/runner/typescript/live-runner.test.ts:203">
P2: Use Vitest's skip API here; rejecting the async test will mark missing fixtures as failures instead of skips.</violation>
<violation number="2" location="conformance/runner/typescript/live-runner.test.ts:284">
P1: Extras reporting does not work for array responses, so list endpoints won't surface additive wire fields.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e243c17ef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
63205bf to
7e9c434
Compare
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="conformance/runner/typescript/schema-validator.ts">
<violation number="1" location="conformance/runner/typescript/schema-validator.ts:163">
P2: `collectExtras` resolves `$ref` only one level, so alias chains (common in response schemas) cause valid fields to be misreported as extras.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
0e243c1 to
bc9c7e2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc9c7e2b92
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 01ea18612f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ebaac5db99
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
5 issues found across 15 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/SchemaWalker.kt">
<violation number="1" location="kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/SchemaWalker.kt:59">
P2: Include OpenAPI wildcard success code `"2XX"` in the fallback response-schema lookup; otherwise valid schemas can be missed and live responses go unvalidated.</violation>
</file>
<file name="conformance/runner/go/replay_runner.go">
<violation number="1" location="conformance/runner/go/replay_runner.go:252">
P2: Validate that each snapshot’s `operation` matches the fixture test’s expected operation before decoding. Without this check, swapped/mislabeled snapshot files can be decoded against the wrong schema and mask fixture mapping errors.</violation>
</file>
<file name="conformance/runner/ruby/replay-runner.rb">
<violation number="1" location="conformance/runner/ruby/replay-runner.rb:34">
P2: `SDK_DECODE` returns `nil` (success) for empty `body_text`, but the real SDK path calls `JSON.parse(body_text)` which raises `JSON::ParserError` on empty strings. This mismatch means the replay runner treats empty-body snapshots as successful decodes that would actually fail through the real SDK, producing false negatives in compatibility checks. Remove the early return or raise on empty body to match real SDK behavior.</violation>
<violation number="2" location="conformance/runner/ruby/replay-runner.rb:75">
P2: Validate that each snapshot file’s `operation` matches the fixture test’s expected operation; otherwise swapped/mislabeled snapshot files can be silently accepted and validated against the wrong schema.</violation>
</file>
<file name="conformance/runner/python/replay_runner.py">
<violation number="1" location="conformance/runner/python/replay_runner.py:163">
P2: Python's `or` treats empty string `""` as falsy, so when `bodyText` is empty this falls through to `json.dumps(page["body"])`. If `body` is `None`, that produces `"null"`, which `json.loads("null")` parses successfully — masking an actually-empty wire body as a successful decode. Compare with the Go equivalent which guards with `bodyText == "" && page.Body != nil`. Use an explicit `None` check instead of `or`:
`body_text = page.get("bodyText") if page.get("bodyText") is not None else json.dumps(page["body"])`
or handle empty bodyText as a decode boundary separately.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
ebaac5d to
149ec69
Compare
Both come in transitively via openapi-typescript → @redocly/openapi-core (fast-uri through @redocly/ajv, brace-expansion through minimatch). npm audit on the affected versions reports: fast-uri ≤3.1.1 — high — path traversal + host confusion (GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc) brace-expansion 4.0.0–5.0.4 — moderate — zero-step sequence hangs the process (GHSA-f886-m6hf-6m8v) Lift both to the patched majors via overrides (alongside the existing minimatch pin). The override path is preferred over plain dependency bumps because the SDK doesn't depend on either package directly — the responsibility for forwarding the fix sits with this repo until the upstream packages cut new releases that pull the patches in transitively. Lands at the bottom of the BC5 stack so the wire-replay PRs above (#294, #301) inherit the fix on rebase.
|
Rebased onto the rebased #293 (
Conformance TS suite green locally: 84 passed / 6 skipped (mock baseline unchanged). |
Both come in transitively via openapi-typescript → @redocly/openapi-core (fast-uri through @redocly/ajv, brace-expansion through minimatch). npm audit on the affected versions reports: fast-uri ≤3.1.1 — high — path traversal + host confusion (GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc) brace-expansion 4.0.0–5.0.4 — moderate — zero-step sequence hangs the process (GHSA-f886-m6hf-6m8v) Lift both to the patched majors via overrides (alongside the existing minimatch pin). The override path is preferred over plain dependency bumps because the SDK doesn't depend on either package directly — the responsibility for forwarding the fix sits with this repo until the upstream packages cut new releases that pull the patches in transitively. Lands at the bottom of the BC5 stack so the wire-replay PRs above (#294, #301) inherit the fix on rebase.
dce5610 to
839b690
Compare
1a40114 to
7c615c3
Compare
|
Rebased onto the new #293 head (839b690, itself freshly rebased onto main@558cfdc3). Rebase-only — no content changes. Sole conflict was the Makefile |
7c615c3 to
cb47f9e
Compare
|
Follow-up: #293 advanced again to 12c0f0a (fast-uri/brace-expansion/js-yaml override bumps for fresh advisories), so this branch is now rebased onto that head — zero conflicts. One new commit rides along: |
* spec(bc5): forward-compat additions + parallel BC4/BC5 provenance
Add optional fields to existing Smithy structures for BC5 read paths,
keeping BC4 readers fully compatible. All additions are opt-in: absent
fields are tolerated, present fields decode through the existing types.
* Notification: bubble_up_url, bubble_up_at (eligibility-gated)
* GetMyNotificationsOutput: bubble_ups, scheduled_bubble_ups; document
current memories[] behavior pending BC team's back-compat decision
* Person: tagline (alias of bio in BC5)
* Todo: steps (reuses existing CardStep wire shape)
* Todoset: todos_count, completed_loose_todos_count, todos_url, app_todos_url
Restructure spec/api-provenance.json for parallel branch tracking — .bc3
keeps the active-branch baseline (now five), .compatibility.bc3-master
records the BC4 baseline for compatibility tracking. Existing tooling
(sync-spec-version, sync-api-version, Go provenance.go) reads the same
.bc3 keys and works unchanged.
Seed spec/bucket-scoped-allowlist.txt with the only existing bucket-only
list endpoint (webhooks) for the new parity lint.
* build(make): add generate aggregator, parity lint, branch-aware sync-status
* make generate — single command to regenerate every machine-derived
artifact in the repo (Smithy, behavior model, URL routes, provenance,
per-language SDKs) in dependency order.
* make check-bucket-flat-parity — new lint that scans openapi.json for
GET /{accountId}/buckets/{bucketId}/<resource>(/...).json list
operations and verifies each has a flat counterpart at
/{accountId}/<resource>.json (or is justified in
spec/bucket-scoped-allowlist.txt). Cross-project SDK consumers
shouldn't have to walk every project to query account-wide data.
* make sync-status — reads .bc3.branch and reports drift for the active
branch (now five) and the compatibility branch (master) as two
clearly-labeled blocks via scripts/report-bc3-drift.sh, instead of
silently mixing them under HEAD.
* make validate-api-gaps target wired into make check (script lands in
the api-gaps registry commit).
* docs(api-gaps): API gap registry + SDK↔BC3 coordination + validator
The SDK is the client-side demand signal for BC3 API coverage. spec/api-gaps/
is a durable, schema-validated registry: each entry tracks one BC5
user-visible feature that ships without (or with incomplete) JSON API
coverage, paired with the BC3 parity plan that owns server-side delivery.
Status changes flow through git history, making the absorption journey
publicly auditable.
Initial registry (11 entries aligned with BC3 plan Phase 3 deliverables):
* calendar — show/update only
* scratchpad — URL path deferred by BC3 plan
* step-top-level — doc-only addition (existing partial)
* everything-aggregates — Phase 3c flat top-level recording listings (~22)
* activity-timeline — /activity (global) + /projects/:id/timeline
* recordable-subtypes-doc — Journal, CloudFile, GoogleDocument; Door excluded
(read paths + create operations per BC3 Phase 3a)
* stack-doc-and-smithy — full CRUD + list collectables
* search-filter-additions — additive filter params
* rich-text-project-attachable — Project as Attachable
* recording-bubbleupable-field — additive boolean
* todoset-completed-list-visibility — pending BC3 classification
Tooling:
* spec/api-gaps/schema.json + allowlist-schema.json validate frontmatter
* allowlist.yml records routes that don't warrant a registry entry
(transient nav state, duplicates already covered elsewhere)
* scripts/validate-api-gaps.{sh,rb} — Ruby stdlib only, no gem deps
COORDINATION.md (root) describes the SDK ↔ BC3 plan relationship and the
absorption lifecycle for anyone reading the SDK repo cold.
CONTRIBUTING.md picks up the api-gaps section, the parity lint section,
and points at make generate as the single command to regenerate
everything.
* ts: regenerate from BC5 spec additions
* ruby: regenerate from BC5 spec additions
* python: regenerate from BC5 spec additions
* kotlin: regenerate from BC5 spec additions
* swift: regenerate from BC5 spec additions
* go: regenerate from BC5 spec additions
* typescript: pin fast-uri and brace-expansion via overrides
Both come in transitively via openapi-typescript → @redocly/openapi-core
(fast-uri through @redocly/ajv, brace-expansion through minimatch).
npm audit on the affected versions reports:
fast-uri ≤3.1.1 — high — path traversal + host confusion
(GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc)
brace-expansion 4.0.0–5.0.4 — moderate — zero-step sequence
hangs the process (GHSA-f886-m6hf-6m8v)
Lift both to the patched majors via overrides (alongside the existing
minimatch pin). The override path is preferred over plain dependency
bumps because the SDK doesn't depend on either package directly —
the responsibility for forwarding the fix sits with this repo until
the upstream packages cut new releases that pull the patches in
transitively.
Lands at the bottom of the BC5 stack so the wire-replay PRs above
(#294, #301) inherit the fix on rebase.
* spec: address Copilot review threads on bucket-parity lint + brief
scripts/check-bucket-flat-parity.sh — drop `--slurpfile spec` and read
the spec via positional input instead. The previous form passed the
spec to jq twice (once via --slurpfile, once positionally), so jq
parsed it twice per invocation. Reading once via positional input then
binding `. as \$s` for cross-iteration access preserves the same
schema-resolution behavior with a single parse.
spec/api-gaps/todoset-completed-list-visibility.md — rewrite the
"PR 1 must classify" sentence so the constraint reads on its own,
without requiring the umbrella-plan cross-reference. The brief now
notes that classification must come from the BC3 side and the brief
stays in the registry either way; readers no longer need to know
which PR "PR 1" referred to.
* make: iterate compatibility provenance blocks generically in sync-status
The sync-status target hardcoded the .compatibility.bc3-master key, which
broke when the post-launch repin renamed the BC4 baseline block to
bc3-four (BC4 now lives on the four branch; master is BC5). Loop over
.compatibility | keys[] instead so any set of compatibility baselines
gets its own labeled drift report without further Makefile edits.
* regen: reconcile generated artifacts after rebase onto main
make generate over the hand-merged spec picks up both main's project
schedule dates (#302) and this branch's BC5 forward-compat additions,
and stamps the api-provenance repin (bc3 = master @ 415b54cc, BC4
compatibility baseline = four @ 9d73959a) through the Go provenance
mirror. go.work.sum picks up entries from the go generate dep download.
* typescript: bump fast-uri, brace-expansion, js-yaml overrides for fresh advisories
Three high-severity advisories published since main's last green security
run (2026-07-20) trip npm Audit on the TypeScript SDK:
- fast-uri GHSA-4c8g-83qw-93j6 (host confusion via failed IDN
canonicalization): override >=3.1.2 -> >=3.1.3, resolves 4.1.1
- brace-expansion GHSA-3jxr-9vmj-r5cp (DoS via exponential expansion):
override >=5.0.6 -> >=5.0.7
- js-yaml GHSA-52cp-r559-cp3m (quadratic CPU via merge-key chains):
new override ^4.3.0, pinned to the 4.x patched line since js-yaml 5
is a major bump for openapi-typescript's transitive use
npm audit clean; ts-generate + ts-generate-services + ts-check verified
green under the new resolutions.
* make, scripts: address review threads on sync-status + parity lint
- sync-status: pass BC3_REPO through to report-bc3-drift.sh in both the
primary and compatibility invocations — make variables aren't exported
to recipe environments, so 'make sync-status BC3_REPO=org/fork' was
silently querying the default repo
- parity lint: use [[:space:]] instead of the non-POSIX \s escape in the
allowlist filter; accept {projectId} alongside {bucketId} in the
bucket-path pattern (and strip logic) for forward-compat with the
documented Smithy path convention; use # as the sed delimiter to keep
the ERE alternation unambiguous across GNU/BSD sed
Candidate count and lint verdict unchanged (verified against openapi.json
before/after).
* make, scripts: second review round on sync-status + parity lint
- parity lint: accept literal filter segments (e.g. /todos/completed.json)
in addition to path-parameter variants — the header always promised
(/<filter>)? coverage but the regex only matched /{param}. Candidate
set unchanged today (verified: single-resource GETs carry no .json
suffix in this spec, so the anchor keeps the match tight).
- sync-status: guard on jq alongside the existing gh guards.
* mode: "mock" | "live" (default mock) * mockResponses required when mode=mock; forbidden when mode=live * liveAssertions[] with liveCallSucceeds, liveResponseFieldsRequired, liveResponseFieldsExpected, liveSchemaValidate * fixtureIds[] for placeholder resolution (PROJECT_ID, TODOSET_ID, etc.) Initial live fixture (live-my-surface.json) covers the read surface exercised by the BC5 forward-compat additions: ListProjects, GetProject, GetMyAssignments + completed + due variants, GetMyNotifications, GetMyProfile, GetTodoset, ListTodolists, ListTodos. Live tests are gated by BASECAMP_LIVE=1 and never reach MSW in mock-mode runs (per the runner's mode filter, separate commit).
The TypeScript runner becomes the canonical wire-capturer for the SDK's compatibility canary: it dispatches each live test through the SDK's typed surface, intercepts the raw HTTP response (bytes + headers) at the global fetch chokepoint, and validates the body against the OpenAPI response schema. Forward-compat fields surface as "extras observed" — never as failures — so new BC5 fields don't break the canary while staying visible. Mode gating * runner.test.ts loads only mode=mock tests; live entries never reach MSW. * live-runner.test.ts loads only mode=live tests; entire suite skips unless BASECAMP_LIVE=1. Missing token/account-id error in beforeAll with a clear message before any test runs. Wire capture * wire-capture.ts wraps globalThis.fetch — single chokepoint for both openapi-fetch (page 1) and the fetchPage closure (page 2+). Cloned responses preserve raw bytes for snapshot persistence; SDK behavior unchanged. * Snapshots persist to <LIVE_RECORD_DIR>/<backend>/wire/<test>.json when the env var is set — feeds cross-language replay (PR 3) and pairwise BC4↔BC5 comparison (PR 4). Schema validation * schema-validator.ts uses ajv + ajv-formats. additionalProperties permissive (forward-compat must not break the canary); required strict; $ref resolved against openapi.json. * Per-page validation; extras union across pages and report in the run-end summary. Fixture-ID resolution * fixtures.ts walks the env-var → discovery → cache ladder per §5d: BASECAMP_BC4_PROJECT_ID / BASECAMP_BC5_PROJECT_ID → BASECAMP_PROJECT_ID → ListProjects → first project. Same pattern for TODOSET_ID, TODOLIST_ID, TODO_ID. Missing-fixture path skips with skipReason. Dispatch coverage gate * live-dispatch.ts maps operation → SDK call. assertDispatchCoverage() fires in beforeAll: any operation referenced in the fixture without a dispatch case fails the run with a clear error pointing at where to add the mapping. Make + docs * `make conformance-typescript-live` opt-in target (not invoked by `make check`). CONTRIBUTING.md documents env vars and the fixture/dispatch pairing rule. Out of scope per PR 2: cross-language wire replay (PR 3), pairwise BC4↔BC5 comparison (PR 4), API gap detector (PR 5).
Three fixes from PR review against the live-only path that offline
checks didn't exercise:
1. Schema validator $ref resolution. The validator registered the full
OpenAPI doc with key "openapi.json" but compiled the response-schema
fragment directly. Refs like "#/components/schemas/ListProjectsResponseContent"
resolved against the fragment root (not the registered doc), throwing
"can't resolve reference ... from id #" on every live call. Fix:
prepareForCompile rewrites local refs ("#/...") to fully-qualified
refs ("openapi.json#/...") so Ajv resolves against the registered
document. additionalProperties:false is still stripped in the same
pass.
2. Vitest-native skip for fixture-missing. The previous code did
Promise.reject(Object.assign(new Error("SKIP..."), { skip: true })),
which Vitest reports as a failure regardless of any custom flag. A
sandbox missing a todoset/todolist would have failed the canary
despite the documented "skip with skipReason" behavior. Fix: use the
test-context's runtime skip — testCtx.skip(reason) — so missing
fixtures land as proper skipped tests.
3. Recursive extras collection. collectExtras() only inspected the root
when it was an object, so list responses (ListProjects, ListTodos,
etc.) yielded no extras even when items had unknown fields. Fix:
walk body + schema together, descend through arrays (path segment
"[]") and into known properties; emit dotted paths like
"[].future_field" or "unreads[].future_envelope_field". Bounded
depth as a cycle guard.
Offline coverage: schema-validator.test.ts exercises validateResponse
with crafted payloads — conformant ListProjects/Project, missing
required field, forward-compat extras at root and item level, plus
GetMyNotifications object-with-array properties. 10 tests, all pass.
This is the regression bar for the live-only path: future schema
wiring bugs surface on every PR via mock conformance, no live creds
needed.
Verified: BASECAMP_LIVE=1 BASECAMP_TOKEN=stub BASECAMP_HOST=<unreachable>
runs report 4 fixture-needing tests as skipped (ctx.skip path) and
6 host-needing tests as expected-failed (network). No false failures.
Four issues from the cubic/codex review pass on the live-only path:
1. Schema lookup ignored non-200 success responses. findResponseSchema()
only checked "200" then "default", missing operations that return 201
(Create*) or 204 (Delete*). Now scans 200/201/202/203/204/default
first, then any 2xx code, before declaring "no schema found".
2. resolveRef only resolved one level. Alias chains
(Foo → $ref Bar → $ref Baz) caused valid item-level fields to be
misreported as extras. Now loops until non-ref or cycle detected.
3. assertDispatchCoverage used `op in LIVE_OPERATIONS`. Inherited
Object.prototype keys (`toString`, `hasOwnProperty`, `constructor`)
would have slipped past the gate. Switched to Object.hasOwn().
4. Wire capture window included fixture-discovery traffic. dispatch(ctx)
would call resolveFixtureId() inside the capture window, mixing the
ListProjects discovery page into the snapshot for GetProject. Split
the dispatch contract into DispatchSpec { fixtures, call }: the
runner pre-resolves all required fixtures before installing capture,
then runs the call() with already-resolved IDs. Discovery happens
strictly before capture starts, so snapshots contain only the
operation under test.
Offline coverage:
* schema-validator.test.ts gains a "non-200 success" case asserting
CreateProject (201) finds a schema (vs the pre-fix "no schema found").
* live-dispatch.test.ts (new) covers the coverage gate including the
Object.hasOwn semantics — fixture entries for `toString` /
`hasOwnProperty` / `constructor` are correctly rejected.
15 offline tests pass; full TS conformance: 78 passed, 6 skipped.
The live-canary fixture (conformance/tests/live-my-surface.json) sits in
the shared conformance/tests directory but only the TS runners know
about the `mode` field. Without this fix:
* Kotlin parses every JSON in the directory; ${PROJECT_ID} placeholders
fail Int conversion and live-only operations are unknown — 9 hard
failures in CI.
* Go has the same loader pattern but its assertion checker silently
ignores entries lacking `assertions` (live entries use `liveAssertions`),
so the 10 live entries slip through as false passes (78/0 vs the
expected 68/0).
* Ruby and Python share the same shape; same risk.
Make `mode` a cross-runner contract: the four offline runners (Go,
Ruby, Python, Kotlin) now filter `mode == "live"` entries at load time.
TS runners already filtered. Schema stays shared in conformance/tests/;
no fixture relocation needed.
Verified with `make conformance` (each runner separately):
* Go: Passed: 68, Failed: 0
* Ruby: 59 passed, 9 SDK-specific skips
* Python: 64 passed, 4 SDK-specific skips
* Kotlin: Passed: 58, Skipped: 10
* TS: 3 mock test files passed, live-runner.test.ts skipped without BASECAMP_LIVE
Each runner now reports 68 total (10 fewer than the pre-fix 78 — exactly
matching the live-only entries that now correctly filter out).
…chema
Ruby was the only offline runner expressed as "reject live" instead of
"accept mock". With the schema enum bounded to {mock, live} the two are
equivalent today, but accept-mock defends against any future mode (e.g.
wire-replay) silently leaking into the offline runner.
Also document in CONTRIBUTING.md that mode is a shared-schema contract
across runners — fixtures live in conformance/tests/ for everyone, so
offline runners must filter to mock before dispatch.
live-runner.test.ts `fieldExists` — when descending into an array, the remaining-path computation used `parts.indexOf(part)`, which returns the first occurrence of that segment name. For required-field paths with repeated names (`a.b.b.x`), recursion restarts from the wrong position and reports false missing-field failures. Iterate by index instead. schema-validator.ts `validateResponse` — when `findResponseSchema` returns null, the validator returned `ok: false` unconditionally. That broke schema validation for legitimately bodyless 2xx operations (e.g. `DeleteBoost` with only a 204 response), blocking them from inclusion in the live canary. Distinguish bodyless-by-design (`ok: true`, nothing to validate) from missing-operation (`ok: false`) via a new `operationHasBodylessSuccessOnly` helper that checks whether the operation declares any 2xx response without an `application/json` schema. New regression test pins the `DeleteBoost` case.
…-uri bump Three review blockers on the live canary: * Wire capture now collapses consecutive captures of the same URL into the final attempt. The SDK's retry middleware re-issues the identical request through global fetch on 429/503, so a recovered call recorded the failed attempt as pages[0] — failing liveCallSucceeds despite the call succeeding, and polluting pages_count/pairwise comparison with retry noise. Distinct pagination pages keep distinct URLs, so only retry attempts collapse. Covered at the capture layer directly and through the real retry middleware (mocked 429 → 200; one page, status 200, calls seen = 2). * validateResponse no longer waves through a present body on operations whose spec declares only bodyless 2xx responses. A payload where the spec promises none is contract drift; succeeding masked 204-style operations growing undocumented bodies. Absent bodies (null/undefined/ empty string) still pass. Both directions unit-tested. * conformance runner lockfile: fast-uri 3.1.0 → 3.1.3 (transitive via ajv) clearing GHSA-q3j6-qgpj-74h6 / GHSA-v39h-62p7-jpjc, which failed the Dependency Review check.
npm audit --audit-level=high flagged fast-uri <=3.1.3 (host confusion via literal backslash authority delimiter). npm audit fix resolves 3.1.4 within ajv's ^3.0.1 range; audit is clean. Companion to the SDK-side override bumps on the spec-foundations branch.
cb47f9e to
9a6f062
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a6f062879
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| capture.restore(); | ||
| } | ||
|
|
||
| const snapshot = capture.drain(); |
There was a problem hiding this comment.
Verify live captures against fixture routes
After draining the capture, the live runner starts evaluating assertions without checking the captured request URL/method against tc.path/tc.method; this already lets the new GetProject and GetTodoset fixtures declare ...${ID}.json paths even though the generated clients dispatch /projects/{projectId} and /todosets/{todosetId} (no label suffix). A live run can therefore pass while snapshotting a different route than the fixture claims, so please resolve the fixture placeholders and assert the captured route (or remove/correct the unused path metadata).
AGENTS.md reference: AGENTS.md:L178-L183
Useful? React with 👍 / 👎.
…reate-bucket-scope-20260601145434 * origin/main: (28 commits) Upstream sync: repin provenance to bc3 ba105ba7 (2026-07-22), register post-train contracts, codify pin semantics (basecamp#390) ci: pin conformance uv sync to the matrix Python; parse the fan-in gate (basecamp#388) BC5 readiness: pairwise BC4↔BC5 + check-bc5-compat orchestrator + scheduled CI (basecamp#308) CI: fold conformance suites into their language test jobs (basecamp#387) Fix TypeScript SDK Content-Type on GET requests (basecamp#384) BC5 launch reconciliation: registry refresh for the shipped BC5 API train (basecamp#325) BC5 readiness: cross-language wire-replay decoders (basecamp#301) BC5 readiness: Go wrapper forward-compat audit + field-level drift check (basecamp#309) BC5 readiness: TypeScript live wire-capture canary (basecamp#294) BC5 readiness: forward-compat spec, briefs, parity lint (basecamp#293) deps(ts): bump brace-expansion from 5.0.2 to 5.0.7 in /typescript (basecamp#386) typescript: bump fast-uri, brace-expansion, js-yaml overrides for fresh advisories (basecamp#385) deps(actions): bump the actions group with 6 updates (basecamp#382) deps(ruby): bump the bundler-dependencies group in /ruby with 2 updates (basecamp#379) deps(kotlin): bump the gradle-dependencies group (basecamp#383) deps(go): bump github.com/oapi-codegen/oapi-codegen/v2 in /go (basecamp#381) deps(ts): bump the npm-dependencies group in /typescript with 2 updates (basecamp#380) deps(smithy): bump the smithy-gradle-dependencies group (basecamp#378) deps(go): bump github.com/oapi-codegen/runtime (basecamp#377) Todos Update/Edit/Replace: merge-safe writes across all 6 SDKs (basecamp#375) ... # Conflicts: # go/pkg/basecamp/api-provenance.json # go/pkg/basecamp/version.go # go/pkg/generated/client.gen.go # kotlin/sdk/src/commonMain/kotlin/com/basecamp/sdk/BasecampConfig.kt # openapi.json # python/src/basecamp/_version.py # ruby/lib/basecamp/generated/metadata.json # ruby/lib/basecamp/generated/types.rb # ruby/lib/basecamp/version.rb # spec/api-provenance.json # spec/basecamp.smithy # swift/Sources/Basecamp/BasecampConfig.swift # typescript/src/client.ts # typescript/src/generated/metadata.ts # typescript/src/generated/openapi-stripped.json
Summary
The TypeScript runner becomes the canonical wire-capturer for the SDK's compatibility canary. It dispatches each live test through the SDK's typed surface, intercepts the raw HTTP response at the global fetch chokepoint, and validates the body against the OpenAPI response schema.
Strictly the TS layer of the multi-language canary plan: cross-language replay and pairwise BC4↔BC5 comparison are deferred to follow-up PRs.
What lands
conformance/schema.json):mode: "mock" | "live"(default mock);mockResponsesconditionally required;liveAssertions[]with four assertion types;fixtureIds[]for placeholder resolution.conformance/tests/live-my-surface.json): 10 operations covering the read surface BC5 forward-compat additions touch (GetMyNotifications,GetTodoset,GetMyProfile, etc.).runner.test.tsloads only mock tests;live-runner.test.tsloads only live tests, skips entirely withoutBASECAMP_LIVE=1.wire-capture.ts): wrapsglobalThis.fetch— single chokepoint for bothopenapi-fetch(page 1) and thefetchPageclosure (page 2+). SDK behavior unchanged.schema-validator.ts): ajv + ajv-formats.additionalPropertiespermissive (forward-compat doesn't break canary);requiredstrict;$refresolved againstopenapi.json. Per-page validation; extras union across pages and surface in the run summary.fixtures.ts): env-var → discovery → cache ladder (§5d).BASECAMP_BC4_PROJECT_ID→BASECAMP_PROJECT_ID→ListProjects → first project, same forTODOSET_ID,TODOLIST_ID,TODO_ID.live-dispatch.ts):assertDispatchCoverage()fires inbeforeAll. Any operation in the fixture without a dispatch case fails the run with a pointer to where to add the mapping.LIVE_RECORD_DIRis set, per-test snapshots write to<dir>/<backend>/wire/<test>.jsonfor downstream replay (PR 3) and pairwise comparison (PR 4).make conformance-typescript-live: opt-in target; not invoked bymake check.Acceptance bar
Per the scope guard:
BASECAMP_LIVEunset → mock only; set → live only.BASECAMP_BACKEND.Test plan
make conformance-typescript(default, no live env) —live-runner.test.tsskipped,runner.test.ts63 mock tests pass + 5 pre-existing TS_SDK_SKIPS.BASECAMP_LIVE=1 npx vitest run live-runner.test.ts(no token) — fails fast inbeforeAllwith a clear missing-env-var error.NoSuchOperationto the fixture; runner refused to start with:Live runner is missing dispatch cases for: NoSuchOperation. Add a DispatchFn to LIVE_OPERATIONS in live-dispatch.ts.Reverted before commit.BASECAMP_LIVE=1 BASECAMP_TOKEN=... BASECAMP_ACCOUNT_ID=... BASECAMP_HOST=... LIVE_RECORD_DIR=tmp/live-canary/bc4 BASECAMP_BACKEND=bc4 make conformance-typescript-liveagainst a real BC4 sandbox. (Requires sandbox creds — reviewer to confirm or defer to follow-up.)Out of scope (later PRs)
make check-bc5-compatorchestrator + scheduled CI.scripts/detect-api-gaps.sh).Summary by cubic
Adds a TypeScript live wire-capture canary that validates raw API responses against the OpenAPI schema, collapses same‑URL retries, pre‑resolves fixtures, and writes snapshots for replay. Live runs are opt‑in; offline runners load only mock tests.
New Features
mode(mock/live),liveAssertions, andfixtureIdswith mode‑specific requirements; newconformance/tests/live-my-surface.json(~10 read endpoints).globalThis.fetch, validates withajv+ajv-formats, unions extras in the run summary, and persists snapshots toLIVE_RECORD_DIR/<backend>/wire/*.json; gated byBASECAMP_LIVE=1withBASECAMP_TOKEN/BASECAMP_ACCOUNT_ID;BASECAMP_HOSTis origin‑only and the runner appends/{accountId};make conformance-typescript-livetarget and CONTRIBUTING docs added.Object.hasOwn; fixture IDs resolve via env → discovery → cache and are pre‑resolved before capture; Go/Ruby/Python/Kotlin filter tomode: "mock"at load time.Bug Fixes
$reftoopenapi.json#/…, resolves multi‑level$refchains, finds non‑200 2xx schemas (e.g., 201), treats bodyless 2xx (204‑only ops) as success and flags bodies there as drift, and collects extras recursively through arrays/objects.testCtx.skipfor missing fixtures, keeps discovery traffic out of snapshots, and collapses consecutive same‑URL retries so only the final attempt is recorded.fast-urito 3.1.4 to clear advisories.Written for commit 9a6f062. Summary will update on new commits.